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Abstract 


Real-time  citizen  interaction  has  the  potential  to  transform  society's  response  to  crisis  situations.  New 
systems  and  processes  must  be  developed  to  support  citizen  involvement  and  first  responders  must  be 
trained  in  their  use.  The  Citizens'  Emergency  Response  Portal  System  (CERPS)  Simulation  Experiment 
(CERPS  SIMEX)  evaluated  examined  the  role  of  public  interaction  through  social  media  during 
emergency  situations.  The  SIMEX  was  conducted  over  five  days  in  October  2012  at  the  Net-Centric 
C4ISR  Experimentation  Lab  (NCEL)  housed  at  The  MITRE  Corporation's  McLean,  Va.,  headquarters 
and  on  the  George  Mason  University  (GMU)  campus  in  Fairfax,  Va.  No  actual  emergency  activities 
occurred  on  the  campus;  the  SIMEX  was  conducted  behind  a  firewall  to  avoid  unintentional  public 
panic.  The  SIMEX  brought  together  emergency  response  personnel  from  federal,  state,  county  and  city 
jurisdictions  in  the  National  Capital  Region  of  Washington,  D.C.  Emergency  operators  used  real 
command  and  control  systems  with  simulated  reporting  and  sensor  systems.  Citizen  participants  were 
volunteers  recruited  from  the  GMU  student  body.  The  primary  purpose  of  the  SIMEX  was  to  examine 
the  hypothesis  that  citizen  participation  via  social  media  in  crisis  response  decision-making  can 
improve  the  outcome  of  a  crisis.  This  SIMEX  was  intended  to  establish  a  baseline  for  future  research 
on  crisis  response  decision  support  employing  citizen  participation. 


1.  Introduction 


Crowdsourcing  promises  to  be  a  game-changing  technology  for  crisis  response.  A  recent  survey  by 
the  American  Red  Cross  found  that,  after  television  and  local  radio,  the  Internet  is  the  third  most 
common  source  of  emergency  information  for  the  public  in  crisis  situations  (ARC,  2010).  A  recent 
United  Nations  Foundation  report  states,  'Towered  by  cloud-,  crowd-,  and  SMS-based  technologies, 
individuals  can  now  engage  in  disaster  response  at  an  unprecedented  level...  This  trend  toward 
communications  driven  by  and  centered  on  people  is  challenging  and  changing  the  nature  of 
humanitarian  aid  in  emergencies,"  (OCHA,  2011).  The  report  describes  how  Haitian  citizens  used 
social  media  and  mobile  technologies  to  call  for  help  during  the  2010  earthquake,  and  how 
volunteers  collated  and  analyzed  the  data  to  generate  maps  depicting  regions  in  urgent  need  of  relief. 
Social  media  figured  prominently  in  the  Government  response  to  Hurricane  Irene  in  August  2011 
(Greenberg,  2011).  Nanfuka  (2012)  and  Bharania  (2012)  describes  numerous  grassroots  efforts 
exploiting  the  Internet  and  social  media  to  assist  victims  of  Hurricane  Sandy.  These  and  other 
examples  demonstrate  the  power  of  social  media  and  communications  technology  to  focus  and  target 
crisis  response  efforts. 

To  achieve  the  full  benefit  of  citizen  involvement,  new  systems  and  processes  are  needed  to  exploit 
the  power  of  modern  information  and  communications  technology.  Grassroots  efforts  are  powerful 
and  important,  but  if  they  are  not  properly  coordinated,  overloaded  responders  and  mis-allocation  of 
resources  can  result.  Research  is  needed  to  design,  implement,  evaluate  and  refine  systems  and 
processes  that  effectively  integrate  grassroots  citizen  participation  within  Federal,  State  and  Local 
crisis  response.  With  any  new  technology  come  problems  and  pitfalls  -  some  already  identified  (e.g., 
Lorenz  et  al.,  201 1)  and  others  yet  to  be  discovered.  Experimentation  is  needed  to  discover  points  of 
failure,  build  in  safeguards,  and  capitalize  on  what  works.  Training  is  needed  for  responders  to 
integrate  crowdsourcing  technology  into  crisis  response  plans. 

As  an  early  step  in  this  direction,  MITRE  Corporation  and  George  Mason  University  (GMU),  in 
coordination  with  government  sponsors,  conducted  the  Citizens  Emergency  Response  Portal  System 
(CERPS)  Simulation  Experiment  (SIMEX).  The  CERPS  SIMEX  was  conducted  over  five  days  in  October 
2012  at  the  Net-Centric  C4ISR  Experimentation  Lab  (NCEL)  housed  at  The  MITRE  Corporation's 
McLean,  Va.,  headquarters  and  on  the  George  Mason  University  (GMU)  campus  in  Fairfax,  Va.  No 
actual  emergency  activities  occurred  on  the  campus;  the  SIMEX  was  conducted  behind  a  firewall  to 
avoid  unintentional  public  panic.  The  primary  purpose  of  the  experiment  was  to  examine  the 
hypothesis  that  citizen  participation  in  crisis  response  decision-making  can  improve  the  outcome  of  a 
crisis. 

The  MITRE  Net-Centric  C4ISR  Experimentation  Lab  (NCEL)  has  sponsored  41  SIMEXs  over  the  past 
ten  years.  Each  SIMEX  uses  real  command  and  control  systems,  simulated  reporting  and  sensor 
systems,  and  actual  military  and  civilian  operators  executing  various  crisis  action  scenarios.  The 
SIMEXs  address  interoperability  among  systems  and  support  concept  of  operations  development, 
tactics  development,  and  insertion  of  new  technologies.  This  SIMEX  was  unique  in  its  involvement  of 
ordinary  citizens  as  an  integral  part  of  the  operation.  The  citizen  participants  were  volunteers  from 
the  GMU  student  body  who  monitored  and  participated  in  the  decision  making  process  through 
crowd-sourcing  technology.  The  SIMEX  established  a  baseline  for  future  research  to  examine  the 
causal  factors  leading  to  successful  crisis  response  outcomes.  The  experience  gained  during  the 
SIMEX  will  support  the  design  of  future  tools  and  processes  to  incorporate  crowdsourcing  technology 
into  crisis  response. 
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2.  Citizens  Emergency  Response  Portal  Simulation  Experiment 

The  Citizens  Emergency  Response  Portal  System  Simulation  Experiment  (CERPS  SIMEX)  examined 
command  and  control  processes,  technologies,  decision-making  theory,  and  socio-cultural  issues 
associated  with  employing  CERPS  in  a  military/civilian  contingency  operation.  The  purpose  of  the 
SIMEX  was  to  investigate  the  impact  of  CERPS  on  the  execution  of  that  operation.  The  SIMEX 
employed  a  "dirty  bomb"  scenario  set  in  the  National  Capital  Region  (NCR).  The  simulated 
detonation  occurred  on  the  campus  of  George  Mason  University  (GMU).  The  scenario's  notional 
military  and  civilian  emergency  managers  collaborated  in  response  to  the  event.  Military  and  civilian 
emergency  management  staff  employed  their  respective  command  and  control  systems  in 
conjunction  with  CERPS  to  collaborate  and  make  response  decisions  from  their  respective  operations 
centers,  which  were  emulated  at  NCEL.  NCEL  participants  included  representatives  from  the 
Department  of  Defense,  Law  Enforcement,  and  State,  County  and  campus  emergency  response 
organizations.  First  responder  reporting  systems  and  events  on  the  GMU  campus  were  simulated  via 
the  Emergency  and  Disaster  Management  Simulation.  Volunteers  from  the  GMU  student  body  played 
the  role  of  citizens. 

The  origins  for  the  CERPS  SIMEX  come  from  Presidential  Policy  Directive  (PPD)  8  and  the  National 
Strategic  Narrative.  PPD-8  states  "Our  national  preparedness  is  the  shared  responsibility  of  all  levels 
of  government,  the  private  and  nonprofit  sectors,  and  individual  citizens.  Everyone  can  contribute  to 
safeguarding  the  Nation  from  harm"  (Obama,  2012).  The  National  Strategic  Narrative  reiterates  this 
view  saying,  "...[the  Nation]  requires  a  strong  and  unshakable  economy,  a  more  diverse  and 
deployable  Inter  Agency,  and  perhaps  most  importantly  a  well-informed  and  supportive  citizenry" 
(Porter  and  Mykleby,  2011).  Thus,  there  is  a  solid  basis  in  national  policy  to  examine  the  role  and 
impact  of  citizen  participation  during  a  crisis. 

2.1.  Hypothesis 

The  hypothesis  investigated  by  the  SIMEX  was: 

The  real-time  interaction  of  the  American  citizenry  in  the  planning  and  execution  of  a 
military/civilian  contingency  operation  would  improve  its  result.  A  viable  method  of  including 
American  citizens  in  the  decision  making  process  would  be  through  the  employment  of  a  version 
of  crowdsourcing  technology. 

2.2.  The  Citizens  Emergency  Response  Portal  System 

CERPS  is  a  prototype  set  of  social  media  and  social  media  analysis  tools  developed  to  provide 
crowdsourcing  technology  for  use  during  this  experiment.  When  combined  with  a  virtual  Emergency 
Operations  Center  (EOC),  CERPS  was  provided  a  conduit  for  interaction  between  the  public  and 
emergency  managers.  The  goal  of  the  CERPS  experiment  was  to  understand  the  impact  of  this 
interaction  on  the  public  and  on  emergency  operations. 

The  CERPS  has  six  components: 

1.  CERP,  a  variant  of  the  Ushahidi  crowdsourcing  platform  that  provides  distributed  situational 
awareness  between  citizens  and  the  operations  center  (see  Figure  1  for  a  screen-shot); 

2.  The  Simulated  Sensory  Environment  (SSE),  a  simulation  of  GMU  in  the  virtual  world  of  the 
scenario  (see  Figure  2  for  a  screenshot; 

3.  Chirp,  an  emulation  of  the  twitter  output  from  citizens  and  operations  center; 

4.  The  Social  Radar  Analytical  Toolset  (SRAT)  that  analyzes  and  interprets  citizen  input  from 
CERP  and  Chirp; 
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Figure  1:  Screenshot  of  CERP  (from  Ludwig,  et  al,  2012) 


5.  The  Virtual  Citizen  Operational  Presence  (VCOP)  that  is  the  analytical  output  of  SRAT;  and 

6.  The  evolving  Crisis  Response  Decision  Space  (CRDS)  that  dynamically  portrays  decisions  and 
options  being  considered  by  the  operations  center. 

Citizen  participants  logged  into  CERPS  via  laptops  from  locations  on  or  near  campus  to  comment  on 
their  perceptions  of  the  unfolding  crisis  and  the  planned  response  decisions  depicted  in  CRDS.  The 
experiment  evaluated  ease  of  use,  how  well  CERPS  facilitates  and  encourages  citizen  participation, 
and  how  clearly  and  simply  the  crisis  operation  center's  decision  process  was  depicted  to  the 
citizenry  during  the  crisis. 

2.3.  Objectives 

The  objectives  of  the  CERPS  SIMEX  are  listed  below.  The  primary  and  general  objectives  were  of 
interest  to  all  participants.  Additional  focused  objectives  related  to  more  specific  concerns  of  one  or 
more  groups  of  participants. 

•  Primary  objective.  Examine  the  impact  of  CERPS  and  citizen  involvement  on  tactical  /  oper¬ 
ational  decision-making  and  execution. 

•  General  objectives,  (a)  Observe  state/local  participation/response  and  the  interaction  with 
federal  response;  (b)  Evaluate  the  use  of  federal  level  resources  for  the  radiological  disaster;  (c) 
enhance  awareness  for  a  Radiological  Dispersal  Device  (RDD)  scenario;  and  (c/)  Avoid  public 
panic  by  controlling  the  public  message. 

•  Social  media  objectives,  (a)  Observe  integration  and  sharing  of  crowd-sourced  social  media 
information  with  local  and  regional  first  responders,  decision  makers  and  leadership;  (b) 
Determine  how  to  layer  the  use  of  crowdsourcing  on  top  of  current  operations;  (c)  Observe 
how  operators  fuse  social  media  indicators;  and  (d)  Assess  the  utility  of  social  media 
information  (CERP,  Chirp,  CRDS,  polls,  and  VCOP)  for  local/regional  emergency  operations 
and  the  military  staff  assessment  process. 
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Visualization: 

•  A  live  video  stream  of  the  3D 
virtual  environment 

•  A  2D  map  with  notable  locations 

•  A  virtual  camera  to  take  pictures 
and  share  with  others 


Simulation  of  other  senses: 

Descriptions  of  auditory,  tactile  * 
and  olfactory  sensations 


Figure  2:  Simulated  Sensory  Environment  (from  Ludwig,  et  al,  2012) 


•  Federal  agency  objectives,  (a)  Assess  NCEL  experimentation  methodology  for  use  in  National 
Emergency  Management-driven  exercises;  (b)  Support  a  simulation  cell  in  the  EOC  for 
interagency  use  during  SIMEX;  (c)  Examine  the  value  of  human  simulation  in  exercise  to 
represent  general  public  for  use  in  National  Emergency  Management-driven  exercises;  and  (d) 
Examine  the  logistics  (e.g.,  distributed  data  input  locations)  of  mass  human  simulation  in 
National  Emergency  Management-driven  exercises. 

•  Law  Enforcement  Objectives,  (a)  Determine  the  type  of  information  received  from  citizen 
input  and  how  it  relates  to  investigation;  and  (b)  Observe  public  reaction  to  law  enforcement 
engagement  in  scenario  and  determine  how  to  utilize  this  reaction. 

•  George  Mason  University  Objectives,  (a)  Empower  students  to  be  engaged  in  the  experiment 
and  become  active  SIMEX  participants;  (b)  Protect  the  privacy  and  safety  of  students;  and  (c) 
Obtain  student  feedback  on  value  of  CERPS  from  their  perspective. 

2.4.  Command  Structure 

The  command  structure  for  the  SIMEX  was  based  on  the  National  Incident  Management  System 
(NIMS)  (DHS,  2008)  and  the  Incident  Command  Structure,  shown  in  Figure  3.  The  Incident 
Commander  (1C)  was  in  charge  of  managing  the  on-scene  response  from  a  virtual  Incident  Command 
Post  (ICP).  The  EOC  Manager  held  overall  authority  for  the  event  and  was  responsible  for  multi¬ 
agency  coordination  (McKernan,  201 1). 
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Figure  3.  Incident  Command  Structure  (from  Ludwig,  et  al,  2012) 


Typically,  each  participating  organization  operates  from  its  own  EOC.  For  this  SIMEX,  a  notional 
unified  EOC  was  established  allowing  all  the  organizations  to  be  present  in  a  single  facility.  This 
increased  the  interaction  between  participants  and  may  have  reduced  the  response  times  for 
questions  or  requests.  Figure  4  shows  the  layout  of  the  virtual  EOC  established  for  this  experiment. 
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Figure  4.  EOC  Layout  (from  Ludwig,  et  al,  2012) 
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2.5.  Concept  of  Operations 

The  subset  of  emergency  response  and  recovery  operations  of  particular  interest  to  this  SIMEX 
involves  those  activities  that  occur  within  the  EOC,  in  the  instance  of  a  RDD  incident.  The  specific 
procedures  involving  the  response  to  a  RDD  event  can  be  found  in  FBI/CDC  (201 1 ). 

In  addition  to  their  usual  duties,  the  public  relations  participants  were  the  primary  users  of  CERPS  in 
the  various  command  centers  represented  at  this  SIMEX.  These  participants  utilized  CERPS  in  two 
main  cases:  situation  awareness  and  decision  support.  After  the  start  of  the  experiment,  the  public 
relations  emergency  managers  at  the  local,  state  and  federal  levels  monitored  their  systems  and  lines 
of  communication  for  any  reports.  Reports  from  the  public  were  collected,  aggregated,  and  analyzed 
through  the  Virtual  Citizens  Operational  Presence  (VCOP),  which  public  relations  staff  reviewed  and 
incorporated  into  their  common  operating  picture  of  the  incident.  Public  relations  then  formed  a 
report  on  their  situation  awareness  and  shared  it  with  the  citizens  via  the  Citizens  Emergency 
Response  Portal  (CERP)  or  Chirp,  a  stand-in  for  Twitter. 

During  SIMEX  execution,  public  relations  staff  worked  with  the  emergency  managers  to  determine 
possible  courses  of  action  and  assessed  the  desirability  of  some  of  the  courses  of  action  through  the 
Crisis  Response  Decision  System  (CRDS).  Some  potential  courses  of  action  necessitated  engaging  the 
public  on  options  such  as  the  deployment  of  fire  trucks.  In  some  cases,  the  options  under 
consideration  were  released  as  Chirp  messages  and/or  Polls  to  get  specific  feedback  on  the  course  of 
action  under  consideration.  This  was  done  to  obtain  input  to  narrow  down  the  possible  decisions. 


Figure  5  depicts  the  interactions  between  CERPS  and  the  public  performed  by  the  EOC  staff. 


Figure  5.  EOC  Operational  View  (from  Ludwig,  et  al,  2012) 
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2.6.  Simulated  Public 


One-hundred  forty-eight  GMU  students  were  recruited  to  participate  in  the  experiment.  Their  role 
was  to  represent  the  general  public  as  bystanders  to  simulated  events  taking  place  on  campus.  To 
understand  the  events  going  on  around  them,  the  students  used  the  Simulated  Sensory  Environment 
(SSE),  a  virtual  environment  that  depicted  the  events  taking  place  around  them.  Students  then  had 
several  ways  to  share  and  get  information  about  events.  One  source  of  information  was  the 
Simulated  News  Network  (SNN),  which  contained  video  and  text-based  information.  Another 
method  was  the  Citizens  Emergency  Response  Portal  (CERP).  There,  students  could  create  or  read 
reports  about  ongoing  events  and  see  their  locations  on  a  map.  Finally,  students  could  interact  via 
Chirp,  a  Twitter-like  microblogging  service,  which  had  the  additional  capability  of  allowing  polls  to 
be  created.  On  Chirp,  they  could  read  what  other  users  (students  and  emergency  managers)  were 
saying,  ask  questions,  and  receive  messages  directly  from  the  operators.  Interactions  between  the 
EOC  and  the  simulated  public  are  depicted  in  Figure  5. 

3.  Scenario 

The  focus  of  the  experiment  scenario  was  the  response  to  a  radiological  dispersal  device  (RDD) 
explosion  on  the  GMU  campus.  Additional  events,  such  as  medical  emergencies,  demonstrations,  or 
secondary  explosions  were  added  to  stimulate  reactions  from  the  emergency  managers  and  the 
public. 

3.1.  Initial  Conditions 

At  the  start  of  the  event,  there  was  a  Boy  Scout  Jamboree  taking  place  within  the  county.  This  was 
included  because  it  qualifies  as  a  National  Special  Security  Event  (NSSE),  which  was  necessary  to 
explain  why  the  EOC  was  already  operational  with  state  and  federal  resources  nearby  and  available 
on  short  notice. 

Additionally,  two  events  were  scheduled  to  take  place  on  the  GMU  campus:  a  large  concert  at  the 
Patriot  Center,  and  a  speech  by  a  fictional  controversial  author  named  Simon  Pierce  taking  place  at 
the  Johnson  Center.  Due  to  the  controversy  surrounding  the  author's  planned  speech,  protesters  were 
expected  to  gather  in  front  of  the  Johnson  Center. 

Separately,  the  websites  of  the  County  and  GMU  were  hacked  by  a  fictional  terrorist  group  called  the 
Anti-Pierce  Group.  They  defaced  the  websites  and  posted  threatening  messages  warning  of  violence 
should  Pierce  be  allowed  to  give  his  speech. 

3.2.  Emergency  Events 

The  timing,  location,  and  severity  of  the  events  were  varied  between  runs  to  minimize  effects  from 
learning  the  scenario.  The  typical  sequence  of  events  began  with  a  protest  in  front  of  the  Johnson 
center  that  grew  and  became  violent.  Sometime  later  a  vehicle  based  RDD  would  detonate  on 
campus  causing  initial  blast  casualties,  later  followed  by  casualties  from  radiation  exposure.  During 
this  period  of  time,  the  Incident  Commander  and  the  EOC  would  manage  the  emergency  response 
and  make  plans  for  the  shelter  or  evacuation  of  the  campus,  plus  arrange  for  decontamination  as 
needed.  Once  the  emergency  response  was  in  full  operation,  additional  events  such  as  backpack 
bombs  or  gunmen  on  campus  would  take  place  to  force  the  EOC  to  change  its  plans  for  handling  the 
emergency. 

In  parallel  with  this,  the  EOC  would  be  monitoring  and  interacting  with  the  public  over  social  media. 
In  addition  to  the  physical  events,  false  or  misleading  information  was  injected  into  social  media  by 
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obvious  adversaries  or  individuals  attempting  to  impersonate  official  communications  from 
authorities. 

3.3.  Control  Run 

One  scenario  variant  was  unintentionally  introduced  on  the  morning  of  the  fifth  day  of  the 
experiment.  Due  to  the  actual  presence  of  the  President  of  the  United  States  on  the  GMU  Campus 
that  morning,  the  United  States  Secret  Service  had  requested  that  the  students  refrain  from  sending 
messages  about  a  simulated  emergency  on  campus  in  order  to  avoid  confusion  with  any  real 
emergencies  that  might  occur.  This  led  to  the  introduction  of  a  'control'  run  (or  base  case)  where  the 
emergency  managers  would  respond  to  the  simulated  emergency  in  the  scenario  without  the  benefit 
of  using  social  media  for  situation  awareness  and  decision  support,  which  is  how  they  currently 
respond.  The  use  of  social  media  returned  in  an  afternoon  run,  with  the  same  scenario  events 
repeated. 


4.  Execution 

Planning  for  the  CERPS  SIMEX  began  in  April  2012.  Execution  of  the  CERPS  SIMEX  took  place  from 
1-5  October  2012.  The  CERPS  SIMEX  execution  took  place  between  0900  and  1600  daily.  A  daily 
briefing  was  held  each  morning  and  a  daily  hot  wash  each  afternoon.  A  single  scenario  was 
executed  each  day  with  a  one-hour  pause  for  lunch.  The  scenario  for  the  final  day  ended  30  minutes 
early  at  1 530  to  allow  for  a  longer  discussion  with  the  participants  at  the  conclusion  of  the  event. 

5.  Results 

Results  of  the  CERPS  SIMEX  are  presented  with  respect  to  the  objectives  presented  in  Section  2.3. 

5.1.  Results  for  Primary  Objective 

MOE  I:  Impacts  from  citizen  involvement.  Citizen  involvement  affected  both  the  emergency 
managers  and  the  public.  For  the  decision  makers,  citizen  involvement  provided  additional  first-hand 
situation  awareness,  such  as  the  locations  of  fires  or  pictures  of  suspicious  packages.  Reading  Chirps 
and  CERP  reports  helped  emergency  managers  to  decide  when  and  how  to  provide  information  to  the 
public.  Polling  helped  emergency  managers  understand  the  needs  of  the  public  such  as  the  need  for 
more  information  about  shelter  locations. 

Figure  6  shows  the  responses  from  the  operators  to  the  question  of  how  citizen  interaction  impacted 

their  role.  The  majority  of 
participants  indicated  that  citizen 
interaction  had  a  positive  or  very 
positive  impact  on  their 
performance. 

For  the  public,  interaction  with 
emergency  managers  kept  them 
better  informed  about  the  ongoing 
emergency.  It  also  helped  them  to 
feel  that  their  needs  were  being 
taken  into  consideration  by  the 
emergency  managers,  and  helped 
them  to  feel  more  involved  in 
their  own  circumstances  and  less 
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Figure  6.  Impact  of  Social  Media  and  Citizen  Participation 
(from  Ludwig,  et  al,  2012)  p  g 


like  victims.  Figure  7  shows  the  responses  from  the  students  to  questions  about  the  utility  of  messages 
from  the  EOC  and  whether  they  felt  they  were  making  a  contribution.  The  majority  of  students 
answered  that  they  strongly  agree  or  agree  to  both  questions. 

MOE  2:  Usage  of  CERPS  components.  Eye  tracking  equipment  and  analysis  were  used  to  collect 
information  on  the  usage  of  each  CERPS  component.  Results  indicate  that  Chirp  was  the  most 
frequently  used  tool,  both  in  terms  of  number  and  duration  of  views.  Chirp  received  many  long  and 
short  duration  views.  Short  duration  views  tended  to  indicate  that  the  operator  was  checking  for  new 
posts,  often  after  spending  some  time  on  another  app  or  away  from  the  computer.  Longer  view  times 
are  indicative  of  two  behaviors:  posting  a  message  or  conducting  an  in-depth  review  of  all  posts  to 
see  if  any  citizen  posts  required  a  response. 

MOE  3:  CERPS  improvement  to  decisions.  Observations  and  feedback  from  the  emergency  managers 
indicated  that  information  coming  from  the  public,  such  as  pictures  of  events,  helped  emergency 
managers  to  make  better  decisions  about  their  response  to  certain  events.  Likewise,  results  from 
polling  indicated  in  some  cases  that  information  of  greatest  interest  to  the  students  differed  from  what 
was  initially  assumed  by  emergency  managers. 

Finding.  Based  on  the  usage  levels  of  the  social  media  tools  and  the  emergency  managers' 
assessments  of  the  impact  of  citizen  interaction,  the  primary  objective  is  determined  to  have  been 
achieved. 

5.2.  Results  for  General  Objectives 

General  objective  (a),  to  observe  interactions  between  state,  local  and  federal  responders,  was 
achieved.  General  objective  (b),  to  evaluate  the  use  of  federal  level  resources  for  the  radiological 
disaster,  was  partially  achieved.  The  scenario  used  during  this  experiment  was  not  well  suited  for 
triggering  a  federal-level  incident  response,  due  to  the  limited  duration  of  each  scenario.  Some 
federal  resources  were  used,  and  the  mechanisms  for  requesting  other  federal  resources  were 
discussed.  General  objective  (c),  to  enhance  awareness  for  a  RDD  scenario,  was  achieved.  The  RDD 
scenario  prompted  the  participants  to  review  their  RDD  response  procedures  and  to  familiarize 
themselves  with  the  resources  available  from  the  participating  agencies.  The  consensus  of  a  post¬ 
experiment  discussion  among  the  EOC  participants  was  that  their  awareness  for  handling  RDD 
scenarios  had  been  enhanced.  General  objective  (c/),  to  avoid  the  "Orson  Welles  Effect"  by 
controlling  the  public  mess¬ 
age,  was  achieved.  Numer¬ 
ous  steps  were  taken  in  the 
design  of  the  experiment  to 
ensure  that  the  experiment 
did  not  cause  panic  among 
the  general  public.  Tech¬ 
nical  measures  included  re¬ 
quiring  the  use  of  a  virtual 
private  network  to  control 
access  to  the  CERPS  app¬ 
lications,  and  the  use  of 
Chirp  instead  of  Twitter  to 
reduce  the  chances  of  Tweets 
being  sent  outside  of  the 
experimental  channel.  Non¬ 
technical  measures  included  Figure  7.  Impact  to  Public  from  Interaction  (from  Ludwig,  et  al,  2012) 
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Figure  8.  Chirp  and  Poll  Usage  (from  Ludwig,  et  al,  2012) 


explaining  the  experiment  in  advance  to  news  media  and  inviting  them  to  observe  the  experiment. 
Local  jurisdictions  and  the  Secret  Service  were  informed  of  the  experiment  and  scenario.  Public 
Information  Officers  (PIO)  from  the  participating  organizations  monitored  traditional  and  social  media 
during  the  execution  of  the  experiment  to  detect  and  respond  to  any  unintended  mentions  of  the 
experiment  or  scenario. 

5.3.  Results  for  Social  Media  Objectives 

Social  media  objective  (a),  to  observe  integration  and  sharing  of  social  media  information  among  first 
responders,  decision-makers  and  leadership,  was  achieved.  Figure  8  shows  the  number  of  Chirps  and 
Polls  each  day.  The  totals  for  Day  4  are  lower  due  to  a  technical  outage  during  one  run.  The  totals 
for  Day  5  are  lower  due  to  having  only  a  single  run  with  social  media  on  that  day.  While  there  was  a 
learning  curve  associated  with  use  of  the  new  tools,  most  operators  were  using  CERP  and  Chirp  after 
the  first  day  and  VCOP  and  CRDS  by  the  third  day.  Trust  in  the  citizen  information  was  a  concern, 
but  operators  learned  to  trust  some  citizens  and  identify  some  deceptive  accounts.  The  main  use  of 
social  media  occurred  after  the  main  response  activities  for  the  emergency  managers  were  completed. 
Messaging  was  most  intense  around  times  when  an  emergency  management  decision,  such  as 
identifying  the  locations  for  shelters  or  evacuation  routes,  had  just  been  made. 

Social  media  objective  (b),  to  determine  how  to  layer  crowdsourcing  on  top  of  current  operations, 
was  partially  achieved.  No  activities  became  unnecessary  as  a  result  of  using  social  media.  Social 
media  provided  beneficial  information,  but  also  caused  additional  workload  for  the  emergency 
managers,  particularly  those  in  PIO  roles.  Time  had  to  be  devoted  to  monitoring  social  media, 
verifying  messages,  tracking  threads,  and  crafting  messages  to  provide  or  solicit  information.  This 
was  especially  the  case  for  91 1-like  messages  that  required  immediate  attention. 

Social  media  objective  (c),  observe  how  operators  fuse  social  media  indicators,  was  achieved.  The 
social  media  tools  were  used  quite  effectively.  Emergency  managers  used  Chirp,  VCOP,  and  Tweet 
Explorer  to  understand  Chirps  about  their  organization.  They  looked  for  multiple  messages  on  the 
same  topic  to  gain  confidence,  and  encouraged  the  use  of  specific  hashtags  to  make  finding  these 
correlating  messages  easier.  Using  hashtags  to  gain  traction  was  a  challenge,  and  there  were  often 
multiple  similar  hashtags  for  a  single  subject.  Messages  with  pictures  or  from  trusted  Chirpers  were 
also  used.  Emergency  managers  frequently  correlated  CERP  and  Chirp  messages  with  those  from  the 
Simulated  News  Network  (SNN).  On  occasion,  they  would  interact  directly  with  SNN  to  learn  more, 
or  to  provide  official  information.  Many  emergency  managers  used  social  media  indicators  (emotion, 
sentiment,  influence)  and  investigated  when  they  saw  anomalies.  However,  the  scale  of  the 
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experiment  was  such  that  the 
operators  could  read  nearly 
every  Chirp  directed  to  them. 
This  would  not  be  the  case  in 
an  emergency  scenario  in¬ 
volving  thousands  to  millions 
of  citizens. 

Social  media  objective  (c/), 
assess  the  utility  of  social 
media  information  (Chirp  and 
polls)  for  local/regional  emer¬ 
gency  operations  and  the  mil¬ 
itary  staff  assessment  process, 
was  achieved.  As  shown  in 
Figure  9,  emergency  man¬ 
agers'  self-assessed  utility  of 
the  social  media  tools  was 
fairly  high.  Assessments  from 
data  collectors  also  shows  high  utility  and  comfort  levels,  as  shown  in  Figure  10.  Results  also  show 
that  the  majority  of  operators  had  some  trust  in  the  information  coming  from  social  media.  This 
indicates  their  level  of  confidence  in  their  ability  to  differentiate  between  legitimate  and  deceptive 
social  media  messaging. 

5.4  Results  for  Federal  Agency  Objectives 

Federal  Agency  objective  (a),  assessing  NCEL  experimentation  methodology,  was  achieved  through 
documenting  the  SIMEX  process  and  inviting  Federal  Agencies  to  send  observers  to  the  SIMEX. 
Federal  Agency  objective  (b),  supporting  a  simulation  cell  in  the  EOC,  was  not  achieved  because  the 
RDD  plume  simulation  was  used  only  in  advance  but  not  during  the  execution  phase  of  the 
experiment.  Federal  Agency  objective  (c),  examining  the  value  of  human  simulation  to  represent  the 
general  public,  was  achieved.  The  demographics  of  the  GMU  student  participants  were  broadly 
comparable  to  the  general  public,  with  the  exception  that  the  students  tended  to  be  younger  than  the 


Utility  of  Social  Media  Tools 


Utility  of  Social  Media 
Information  (0  to  5) 


Chirp 


CRDS 


VCOP 


Figure9.  Self-Assessed  Utility  of  Social  Media  Tools 
(from  Ludwig,  et  al,  2012) 


Figure  10.  Comfort  and  Utility  of  Social  Media  Tools  as  Assessed  by  Data  Collectors 

(from  Ludwig,  et  al,  2012) 
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general  public.  EOC  responders  indicated  that  their  interactions  with  the  students  felt  authentic  and 
provided  a  valuable  dimension  to  the  experiment.  All  emergency  managers  rated  the  student 
responses  between  neutral  and  very  realistic.  Federal  Agency  objective  (c/),  examining  the  logistics  of 
mass  human  simulation  in  National  Emergency  Management  exercises,  was  achieved.  This  aspect  of 
the  experiment  required  consideration  in  the  design  of  the  technical  architecture  and  additional  effort 
for  the  technical  team,  due  to  the  need  to  develop  a  synthetic  environment  for  student  use  and  a 
virtual  private  network  infrastructure  isolated  from  the  open  Internet.  Interaction  was  limited  to 
laptops.  Support  for  mobile  devices  would  increase  realism  and  improve  student  access,  but  required 
too  much  additional  effort  to  be  implemented  for  this  SIMEX. 

5.5  Results  for  Law  Enforcement  Agency  Objectives 

Objective  (a),  to  determine  the  type  of  information  received  from  citizen  input  and  how  it  relates  to 
investigation,  was  achieved.  Information  provided  by  the  students  through  Chirp  and  CERP  were 
mainly  categorized  as  reports  of  suspicious  activity.  This  included  suspicious  persons,  packages,  and 
events.  The  information  was  mainly  text-based,  but  sometimes  also  included  images  of  the 
suspicious  item.  These  inputs  were  used  as  additional  information  to  guide  law  enforcement 
investigations,  and  sometimes  resulted  in  agents  reaching  out  to  individual  students  to  follow  up  in 
more  detail.  The  information  received  from  citizen  input  was  typically  treated  by  the  law 
enforcement  agency  in  the  same  way  that  anonymous  tips  would  be  treated. 

Objective  (b),  to  observe  public  reaction  to  law  enforcement  engagement  in  scenario  and  determine 
how  to  utilize  this  reaction,  was  achieved.  Reaction  to  law  enforcement  involvement  was  determined 
based  on  reviews  of  student  Chirp  postings.  Typically,  the  student  reaction  to  law  enforcement 
involvement  was  positive,  especially  when  law  enforcement  personnel  or  vehicles  were  observed 
responding  to  emergency  situations  on  campus  or  participating  in  the  investigations  after  explosions. 
However,  there  were  a  few  instances  of  negative  reactions  to  law  enforcement  involvement  that 
occurred  when  the  students  mistook  armed  law  enforcement  agents  for  terrorists.  These  cases  of 
mistaken  identity  took  place  at  times  of  intense  scenario  activity.  Some  of  the  mistaken  identity  cases 
may  have  been  the  result  of  poor  viewing  angles  or  limited  graphical  resolution  obscuring  the  law 
enforcement  markings  on  the  simulated  agents'  uniforms. 

5.5  Results  for  George  Mason  University  Objectives 

Objective  (a),  to  empower  students  to  be  engaged  in  the  experiment  and  become  active  SIMEX 
participants,  was  achieved.  Levels  of  student  participation  were  high  during  the  experiment.  Daily 
student  participation  averaged  over  80  students.  Total  Chirps  for  the  week  totaled  approximately 
6700.  Hundreds  of  Chirps  were  either  "Liked",  re-Chirped,  or  replied  to  in  each  run. 

Objective  (b),  to  protect  the  privacy  and  safety  of  students,  was  achieved.  MITRE  and  GMU  took 
numerous  steps  to  protect  the  privacy  and  safety  of  students.  Foremost  among  these  was  submitting 
the  CERPS  experiment  to  the  GMU  Institutional  Review  Board  (IRB)  process  for  approval.  In  addition, 
students  participated  in  the  experiment  using  Chirp  user  names  rather  than  their  actual  names.  Only 
the  GMU  experiment  lead  had  access  to  the  mapping  between  student  names  and  user  names.  To 
protect  the  safety  of  the  students,  all  events  were  simulated  and  took  place  in  a  virtual  environment, 
removing  any  possibility  of  physical  harm.  Additionally,  during  the  sign-up  and  training  sessions,  the 
students  were  informed  of  the  types  of  simulated  events  they  would  be  viewing  and  reacting  to  on 
their  computers  and  given  the  opportunity  to  withdraw  from  the  experiment.  Student  surveys 
indicated  that  over  95%  of  the  student  participants  felt  their  privacy  and  safety  were  adequately 
protected. 
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Objective  (c),  to  obtain  student 
feedback  on  value  of  CERPS  from  their 
perspective,  was  achieved.  Figure  11 
shows  the  students'  assessment  of  ease 
of  use  of  CERPS.  Generally,  students 
found  CERPS  to  be  easy  to  use, 
however  they  found  uploading 
pictures  or  videos  to  CERPS  to  be 
more  difficult.  In  comments,  students 
indicated  their  desire  to  have  a 
smartphone  access  to  CERPS  in 
addition  to  the  laptop  access  that  was 
provided.  Figure  12  shows  student 
responses  to  the  question  dealing  with 
usefulness  of  CERPS.  The  majority  of 
students  felt  that  CERPS  kept  them 
informed  about  the  simulated  emergency.  Additionally,  the  majority  also  found  the  interactions  with 
the  emergency  managers  to  be  informative  and  felt  that  their  contributions  and  opinions  were  being 
taken  into  consideration  by  the  emergency  managers.  Figure  13  shows  student  ratings  of  the  quality 
of  the  simulated  environment.  The  students  found  the  scenarios  and  virtual  environment  to  be 
adequate  for  the  purposes  of  the  experiment.  However,  a  significant  number  of  the  students  found 
the  quality  of  the  imagery  and  graphics  they  viewed  to  be  insufficiently  realistic. 

5.6.  Policy  Findings 

For  the  purposes  of  the  experiment,  it  was  assumed  that  future  policy  would  permit  and  enable  the 
types  of  public  interaction  being  explored.  Discussions  with  emergency  managers  during  the 
planning  and  execution  of  the  CERPS  SIMEX  revealed  that  there  are  numerous  policy  challenges  that 
will  need  to  be  resolved  for  such  a  capability  to  become  operational.  Part  of  the  reason  for  this  is  that 
existing  policies  were  developed  prior  to  the  advent  of  social  media  technology  and  have  not  been 
updated  with  it  in  mind.  Some  of  the  main  policy  concerns  are  outlined  below. 

Expectations .  Emergency  managers  expressed  concerns  over  the  expectations  that  would  be  set  if  they 
began  responding  to  Chirps,  especially  ones  seeking  emergency  assistance.  The  concern  was  that 


50.00% 


I  found  it  easy  to  submit  I  found  it  easy  to  submit 
reports  into  the  CERP  pictures  or  videos  (from 
application.  the  sensory 

environment)  into  the 
CERP  application. 


i  Strongly  agree 


Agree 


Neither  agree  nor 


disagree 
I  Disagree 


I  Strongly  disagree 


Not  Applicable 


Figure  1 1 .  Student  Ratings  of  Ease  of  Use  of  CERPS 
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Figure  12.  Student  Ratings  of  Usefulness  of  CERPS  (from  Ludwig,  et  al,  2012) 
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once  they  started  to  do  that,  the  public  would  then  assume  that  they  would  always  monitor  and 
respond  to  calls  for  help  made  through  social  media.  Feedback  from  students  substantiated  this 
concern  as  several  of  them  pointed  out  their  expectation  that  once  emergency  managers  began  using 
social  media  that  they  would  then  always  be  available  through  social  media.  In  spite  of  these 
concerns,  it  was  mentioned  that  social  media  monitoring  may  have  a  role  to  play  in  any  future  911 
system. 

Liability.  Emergency  managers  expressed  concerns  about  liability  issues  arising  from  their  interaction 
with  the  public.  The  biggest  concern  surrounds  giving  advice,  instruction,  or  asking  assistance  of  a 
member  of  the  public  that  ultimately  results  in  their  injury.  This  concern  may  result  in  controls  or 
limits  on  which  personnel  are  allowed  to  interact  with  the  public,  including  rules  or  procedures  for 
these  interactions.  Clarifying  this  issue  would  make  helping  the  public  easier. 

Privacy.  Emergency  managers  also  expressed  concerns  about  privacy  issues  resulting  from  their 
interactions  with  the  public.  Guidance  will  be  needed  on  how  they  can  handle  things  such  as 
personal  information  and  location.  Subscribing  to  or  following  members  of  the  public  on  social 
media  may  also  raise  privacy  concerns.  While  it  is  important  to  develop  appropriate  policies,  the 
policies  dealing  with  privacy  matters  will  likely  be  different  for  those  engaged  in  emergency  response 
versus  those  engaged  in  law  enforcement  and  investigative  activities. 

5.7.  Other  Feedback 

During  and  after  the  execution  of  the  CERPS  SIMEX,  EOC  operators  and  students  provided  numerous 
additional  insights  about  the  subject  matter.  Some  key  observations  are  given  below. 

•  Emergency  management  agencies  should  have  arrangements  with  counterparts  outside  their 
local  area  who  are  able  to  take  over  social  media  duties  in  the  event  of  an  outage  at  the  local 
EOC 

•  Bad  actors  can  monitor  social  media  and  attempt  to  take  advantage  through  the  use  of 
deceptive  accounts  or  false  reporting 

•  Emergency  managers  and  PIOs  of  the  agencies  involved  in  an  emergency  should  coordinate 
pre-planned  messages  and  responses  to  provide  a  consistent  message  to  the  public  and  to 
help  combat  the  spread  of  rumors 
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Figure  13.  Quality  of  the  Virtual  Environment 


Page  14 


•  Creating  hashtags  and  getting  them  to  gain  traction  during  an  emergency  can  be  challenging. 
PIOs  should  consider  establishing  and  publicizing  hashtags  in  advance,  if  possible. 

•  Current  staffing  levels  or  responsibilities  may  need  to  be  adjusted  to  accommodate  the 
interactive  use  of  social  media 

•  Future  EOC  staffs  may  need  to  include  experts  in  social  sciences  to  assist  PIOs  in 
understanding  information  from  social  media  analysis  tools  and  to  craft  the  best  messages. 

6.  Conclusions  and  Recommendations 

The  CERPS  SIMEX  demonstrated  that  citizen  interaction  with  emergency  managers  can  provide 
positive  impacts.  This  included  providing  additional  911  -type  information  about  incidents,  and 
providing  information  about  the  needs  of  the  public.  Sentiment  analysis  of  social  media  traffic  helped 
emergency  managers  understand  the  mood  of  the  public  and  allowed  them  managers  to  adjust  their 
communications  strategies  to  better  respond  to  the  needs  of  the  public. 

The  CERPS  SIMEX  also  highlighted  some  of  the  challenges  that  engaging  in  public  interaction  through 
social  media  will  bring.  This  includes  the  vetting  of  information  for  accuracy,  the  possible  influence 
of  bad  actors,  and  the  potential  for  emergency  managers  to  be  distracted  by  vocal  social  media  users 
to  the  detriment  of  the  overall  emergency  response. 

Based  on  the  execution,  results,  and  feedback  of  the  CERPS  SIMEX,  the  following  recommendations 
are  made  for  consideration  in  any  future  CERPS  experiments. 

1 .  Increase  the  number  and  diversity  of  people  making  up  the  simulated  public.  A  greater 
number  of  participants  would  result  in  even  higher  levels  of  social  media  traffic,  providing  a 
greater  challenge  for  emergency  managers  and  social  media  analysis  tools.  A  greater  diversity 
of  people  includes  a  wider  range  of  ages,  but  also  educational  backgrounds  and  personal 
experience. 

2.  Include  CERT  members  in  the  simulated  public.  Using  trained  CERT  members  may  help  with 
the  challenge  of  populating  crowd-sourced  disaster  maps  such  as  CERP.  Because  CERT 
members  are  known  and  have  training,  they  may  be  able  to  act  as  trusted  agents,  populating 
CERP  with  information  that  may  not  require  further  verification  before  being  acted  upon  by 
emergency  managers. 

3.  Include  a  Facebook-like  application  as  part  of  the  social  media  toolset.  Experience  from 
emergency  managers  during  Hurricane  Sandy  suggests  that  Facebook  may  be  a  more  heavily 
used  application  than  Twitter  in  disaster  situations. 

4.  Add  the  ability  to  move  around  the  virtual  environment.  Giving  the  simulated  public  even  a 
limited  ability  to  move  freely  would  help  them  become  more  immersed  in  the  scenario.  It 
would  also  enable  them  to  comply  or  not  comply  with  instructions  from  emergency 
personnel.  This  dynamism  would  result  in  a  more  realistic  experience  for  both  the  public  and 
the  emergency  managers. 

5.  Include  a  representation  of  a  91 1  -call  center.  Many  of  the  Chirps  contained  messages  that 
would  be  analogous  to  91 1  calls.  Including  a  representation  of  a  future  91 1  call  center  would 
enable  the  exploration  of  concepts  for  handling  these  types  of  messages.  Additionally, 
traditional  911  calls  about  incidents  will  continue  and  not  be  replaced  by  social  media. 
Having  a  better  representation  of  simulated  91 1  calls  would  enhance  realism. 
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6.  Consider  a  large  regional  disaster  for  future  scenarios.  A  large,  regional  natural  disaster  such  as 
a  hurricane  or  blizzard  would  allow  emergency  managers  to  work  through  the  entire 
preparedness,  response,  and  recovery  process.  This  would  allow  the  impacts  of  citizen 
interaction  to  be  examined  for  each  phase. 

7.  Improve  integration  between  WebEOC  and  CERP.  For  emergency  managers,  using  CERP 
typically  meant  manually  copying  information  from  WebEOC  (a  web-enabled  state/local 
crisis-information  management  system  providing  secure  real-time  information  sharing)  and 
pasting  it  into  CERP.  This  redundancy  reduced  the  usage  of  CERP,  and  thus  the  amount  of 
information  coming  from  the  EOC  to  the  public.  Integration  of  data  between  WebEOC  and 
CERP  would  reduce  the  workload  for  emergency  managers  and  potentially  get  more 
information  out  to  the  public. 

8.  Include  social  media  tools  in  use  in  EOCs  or  developed  by  industry.  PIOs  are  using  some 
social  media  tools  such  as  TweetGrid  and  Twitterfall  today.  Including  tools  currently  in  use 
by  PIOs  would  enhance  operational  realism.  In  addition,  any  future  tools  under  development 
on  behalf  of  the  government  for  use  in  EOC  environments  should  be  considered  for  inclusion. 

9.  Consider  modifying  the  length  of  the  SIM  EX.  Federal,  State  and  Local  operators  suggested  that 
a  week-long  experiment  could  be  too  demanding  on  emergency  managers'  schedules. 
Options  discussed  included  reducing  the  length  of  the  experiment  or  establishing  a  rotation  of 
operators  while  maintaining  the  current  SIMEX  length. 

In  summary,  the  CERPS  SIMEX  provided  a  valuable  demonstration  of  the  value  of  citizen  interaction 
during  a  crisis,  highlighted  policy  issues  that  must  be  addressed  as  social  media  become  integrated 
into  crisis  response  command  and  control,  and  identified  areas  for  further  work  to  enable  effective 
use  of  crowdsourcing  for  emergency  response. 
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Background:  Crowdsourcing  and 
Emergency  Response 


Real-time  citizen  interaction  is  transforming  crisis  response 

►  Haitian  citizens  collaborated  with 
volunteers  worldwide  to  map  damage 
during  2010  earthquake 

►  Social  media  figured  prominently  in 
government  response  to  Hurricane  Irene 

►  “Social  media  follow  Hurricane  Sandy's 
destructive  path”  -  USA  Today 


Command  and  control  systems  and  processes  must  exploit 
new  technologies  for  communicating  directly  with  citizens 


►  Research  is  needed  to  design  and  evaluate  new  systems  and 
processes 

►  Operators  must  be  trained  in  the  new  systems  and  processes 


Background:  Policy  Directives 

Presidential  Policy  Directive-8  (PPD-8)  states:  “Our  national 
preparedness  is  the  shared  responsibility  of  all  levels  of  government, 
the  private  and  nonprofit  sectors,  and  individual  citizens.  Everyone 
can  contribute  to  safeguarding  the  Nation  from  harm..” 

National  Strategic  Narrative  calls  for  diverse  and  deployable  Inter 
Agency,  and  a  well-informed  and  supportive  citizenry.  * 

National  Capital  Region  Homeland  Security  Strategic  Plan  calls  for 
sharing  information  needed  to  make  informed  and  timely  decisions; 
take  appropriate  actions;  and  communicate  accurate,  timely 
information  with  the  public. 

Department  of  Defense  Quadrennial  Defense  Review,  dated 
February  2010,  identified  defending  the  homeland  and  support  to  civil 
authorities  as  one  of  6  key  missions  in  which  the  Department  must 
further  rebalance  policy,  doctrine  and  capabilities 


*  Monograph  from  Woodrow  Wilson  Center  for 
3  Scholars  authored  by  former  members  of  the  Joint 


Hypothesis 

•  American  citizen  real-time  interaction  in  the  planning  and 
execution  of  a  military/civilian  contingency  operation 
would  improve  its  result. 

•  A  viable  method  of  including  American  citizens  in  the 
decision-making  process  would  be  the  employment  of  a 
version  of  crowdsourcing  technology. 

•  Testing  the  hypothesis: 

►  Implement  prototype  DSS  using  crowdsourcing  for  citizen 
participation 

►  Simulate  crisis  in  which  civilian/military  emergency  managers  use 
DSS  interact  with  a  cross-section  of  the  American  public 


Background:  SIMEX 


•  MITRE  Net-centric  C4ISR  Experimentation  Laboratory  (NCEL) 

►  Conducts  simulation  3-5  simulation  experiments  (SIMEXs)  per  year  to  examine 
C4ISR  processes  in  support  of  ground,  maritime,  space  and  air  operations 

►  Use  real  operators,  real  C4ISR  systems,  simulated  scenario  and  reports 

►  42  SIMEXs  conducted  since  2002 

•  SIMEXs  support  multiple  sponsors  to  examine: 

►  Tactics,  techniques  and  procedures  (TTPs) 

►  Concept  of  operations  (CONOPS) 

►  Interoperability  requirements 
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Testing  the  Hypothesis:  A  SIMEX  examining 
Citizen  Participation  in  Crisis  Response 

Primary  Goal:  Examine  impact  of  citizen  involvement  on 

tactical/operational  decision-making  and  implementation. 

Objectives: 

►  Refine  and  evolve  CONOPS  and  TTPs  for 
citizen  participation  in  tactical/operational 
planning  and  implementation 

►  Refine  and  evolve  prototype  DSS 

►  Examine  impact  of  DSS  on  tactical/operational  decision-making  &  execution. 

Scenario:  Defense  Support  of  Civil  Authorities 

►  Radiological  Dispersal  Device  detonates  on  George  Mason  University 
campus. 

►  Notional  NCR  military/civilian  emergency  managers  collaborate  from 
Emergency  Operations  Center  (emulated  at  the  NCEL  lab  at  MITRE  McLean) 

►  Student  volunteers  from  George  Mason  University  use  DSS  to  collaborate  in 
.^sponse  decision  making. 
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Simulated  Sensory 
Environment  (SSE) 


Citizens’  Emergency 
Response  Portal  (CERP) 


Unfolding 
experiment 
events 
(view) 


Reported 
events 
(view  and  post) 


/2 


|  R040  •tOC«>Ol 
|  WWW  OUUM 


Citizens  k 
(GMU  students)  \ 


Chirp  [open-source  Twitter  clone] 


News 


Simulated 
News  Network 


Discussion 
of  events 
(view  and  post) 


“Chirps” 


GEORGE 


UNIVERSITY 


Polling 


C4I  Center 


Citizens’  Emergency  Response 

Portal  (CERP) 

Based  upon  Ushahidi  platform 


Geographic  display  of  incident  reports  and  a 
means  to  review  submitted  reports 

Operators  can  post  directly  to  CERP  to 
provide  official  information 

Operators  view  reports  posted  by  citizens 

Reports  (from  the  map,  listed  In  clironologlcal  order) 


TtTL* 


Pel an  Ten*  Macke?  Oncrge  Uasor  BKd 

Power  ouugo  an  K/iaj  HjI 

Pcww  outage  at  dormtory 


LOCATION  OATC 


Pert*?  Dec*  JMJ20U 


QMU  Cerpui  Jt 
Krug  Hal 

Dor**  I  Jt 


NMrebt 


vie  More 


oot:  ramcipams  viewoT 

o  ~  ~ :  ~ 


+  http*  /  urn  m.tr»  org  tod«a  him 


G- 


Visualization: 

•  A  live  video  stream  of  the  3D 
virtual  environment 

•  A  2D  map  with  notable  locations 
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•  A  virtual  camera  to  take  pictures  | 
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CERPS  SIMEX 


Objective: 

►  Examine  impact  of  CERPS  and  citizen  involvement  on 
tactical  /  operational  decision-making  and  execution 

Participants: 

►  Emergency  management  personnel:  national  (DoD,  FBI,  DHS, 
National  Guard),  state  ,  county,  city,  and  university 

►  GMU  student  volunteers  to  play  role  of  citizens 

Experiment: 

►  Simulate  crisis 

►  Execute  crisis  procedures 

►  Students  interact  with  responders  via  CERPS 

Evaluation: 

►  Examine  results  on  metrics  of  interest 
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CERPS  SIMEX  Operational  View 
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C4I  Center 


SIM  EX  Process 


Experimentation  plan;  Data 

Experiment  data; 

Collection  and  Analysis 

Daily  After  Action  Reviews 

Plan  (DCAP);  Simulation 

architecture;  Network 

architecture 

Concept  paper; 
Scenario  event  list; 
Sequence  diagram; 
Scenario  walkthrough 
questions/issues/gaps; 
Scenario  process  model 


Test  plans; 

Test  reports; 

Problem  report  tracking/resolution 


Quick  Look  Report 
Final  Report; 
Sponsor  briefs 


<^>  -  Initial  Planning  Conference  (IPC) 
<Q>  -  Final  Planning  Conference  (FPC) 


GMU  Tasks 


Advise  on  CONOPS 
Coordinate  IRB  approval 
Recruit  student  participants 
Support  training 


Coordinate  strategic  communications  plan 
with  MITRE  community  relations  (avoid  “war 
of  worlds  effect”) 


Participate  in  EOC 
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Student  Participation 

Participants 

►  Goal:  200  student  participants 

►  Actual:  199  recruited,  125  trained,  114  participated 

►  Paid  $95  in  Mason  money  plus  iPad  for  top  performer 

Activities: 

►  Training  session  (2  hrs) 

►  Test  runs  (2  hrs) 

►  Experimental  sessions  (at  least  5  hrs) 

-  A  different  virtual  emergency  each  day  for  5  days 

-  Respond  to  virtual  environment  through  CERPS 

-  Minimum  of  5  hours 

►  Feedback  session  (no  more  than  30  min) 


Government  Stakeholders 

•  NORTHCOM 

•  Joint  Staff 

•  Fairfax  County 

•  Virginia  Commonwealth 

•  DHS/FEMA 

•  National  Guard  Bureau 

•  FBI 

•  Israeli  Home  Front  Command* 

C4I  Centk 
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Timeline 


•  Summer  201 2: 

►  Develop  concept  of  operations,  scenario, 
data  collection  and  analysis  plans 

►  Obtain  IRB  approval 

►  Develop  publicity  plan 


•  September  2012: 

►  Recruit  and  train  participants 

•  October  201 2: 

►  Conduct  SIMEX  (Oct  1  -5) 

►  Produce  quick-look  briefing 

•  November  2012: 


►  Release  report  to  public 

I 


C4I  Center 


university 
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Results:  Impact  &  Usage 


Responses 


Very  Negative  Neutral  Positive  Very 
Negative  Positive 


■  Responses 
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I  found  the  Chirp 
messages  from  the 
Emergency  Operations 
Center  (EOC)  to  be 
useful. 


I  felt  that  decision 
makers  were  taking  my 
contributions  into 
account. 


■  Strongly  agree 


■  Agree 

Neither  agree  nor 
disagree 

■  Disagree 


■  Strongly  disagree 


■  Not  Applicable 


Figure  6.  Responses  from  Operators: 
Impact  of  Social  Media  and  Citizen  Participation 


Figure  7.  Responses  from  Students: 
Impact  to  Public  from  Interaction 


Results:  Utility  &  Usability 


Utility  of  Social  Media  Tools 
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CERP  Chirp  CRDS  VCOP 


■  Utility  of  Social  Media 
Information  (0  to  5) 
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I  found  it  easy  to  submit  I  found  it  easy  to  submit 
reports  into  the  CERP  pictures  or  videos  (from 
application.  the  sensory 

environment)  into  the 
CERP  application. 


■  Strongly  agree 


Agree 


Neither  agree  nor 
disagree 

■  Disagree 


■  Strongly  disagree 


■  Not  Applicable 


n  c  ..  /  c  u  a  i,,.-,..  Figure  11.  Student  Ratings  of  Ease  of  Use  of  CERPS 

Figure  9.  Emergency  Managers  Self-Assessed  Utility  °  ° 

of  Social  Media  Tools 


Figure  10.  Comfort  and  Utility  of  Social  Media  Tools  as  Assessed  by  Data  Collectors 


Student  Ratings:  Usefulness  and  Quality 
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The  CERP  application 
kept  me  informed 
about  the  simulated 
emergency. 


I  communicated 
several  thoughts 
through  the  Chirp 
application. 


I  found  the  Chirp 
messages  from  the 
Emergency 
Operations  Center 
(EOC)  to  be  useful. 


I  felt  that  decision 
makers  were  taking 
my  contributions 
into  account. 


■  Strongly  agree 
Agree 

Neither  agree  nor  disagree 

■  Disagree 

■  Strongly  disagree 
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Figure  12.  Student  Ratings  of  Usefulness  of  CERPS 


UNIVERSITY 
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The  scenarios  seemed 
believable  to  me  -  I  feel 
that  they  could  have 
actually  happened. 


The  pictures  of  the 
simulated  emergency  were 
convincing 


The  descriptions  that  came 
with  the  pictures  of  the 
simulated  emergency 
allowed  me  to  understand 
what  I  was  seeing,  hearing, 
feeling,  and  smelling. 


■  Strongly  agree 
Agree 

Neither  agree  nor  disagree 

■  Disagree 

■  Strongly  disagree 
Not  Applicable 


Figure  13.  Quality  of  the  Virtual  Environment 
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Media  Attention 

Experiment  Crowdsources  Public  in  Emergency  Response 
Decision-Making 

►  http://www.hstodav.us/industrv-news/qeneral/single-article/experiment- 

crowdsources-public-in-emerqencv-response-decision- 

making/9e632d951  b75fa299ac746a4ce2d55df.html 

This  is  just  a  test:  Emergency  responders  tap  the  Twitterverse 

►  http://www.nextqov.com/emeraing-tech/2012/10/iust-test-emerqencv- 

responders-tap-twitterverse/58622/?oref=na-HPtopstorv 

Mason  Students  Observe  and  Report  During  Mock  Attack  in 
Fairfax 

►  http://about.qmu.edu/mason-students-observe-and-report-durinq-mock- 

attack-in-fairfax/ 

Safety  Tweet:  Northern  Virginia  Magazine  by  Jenna  Makowski 
January  14,  2013 

►  http://www.northernvirqiniamaq.com/buzz-bin/2013/01/15/safetv-tweet/ 


Conclusions 

Demonstrated  potential  for  positive  impacts  from 
citizen  interaction  with  emergency  managers 

►  Augment  91 1  -type  information  about  incidents 

►  Sentiment  analysis  of  social  media  traffic 

Helped  emergency  managers  understand  mood  of  public 

-  Allowed  managers  to  adjust  communications  strategies  to  better  respond  to 
needs  of  public 

Highlighted  challenges  of  public  interaction  through 
social  media 

►  Vet  information  for  accuracy 

►  Account  for  possible  influence  of  bad  actors 

►  Mitigate  potential  for  emergency  managers  to  be  distracted 
by  vocal  social  media  users 


Proposed  SIMEX  14-1 

Maintain  theme 


Include  additional  stakeholders 

Follow  similar  planning  and  execution  schedule 

Incorporate  alternative  tools  as  appropriate  from 
government  and  industry 


Expand  to  include  GMU  campus  and  surrounding 
region  (“College  Town  USA”) 


►  Larger  population  sample 


►  Students,  staff  and  faculty 

►  Other  participants  from  community 

Expand  /  revise  EOC  staffing 


C4I  Center 
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Next  Steps 


The  cloud  and  social  media  bring  major  new 
opportunities  for  decision  support 

CERPS  SIMEX  was  an  important  first  step 

Many  open  research  issues 

►  Effective  integration  of  citizen  input  into  C2  processes 


►  Logistics 

►  Human  factors 

►  Information  security 

Additional  simulation  experiments  are 
proposed 


Thank  You! 


GMU 

•  Stu  Wharton  (participant 
coordinator) 

•  Dave  Farris  (emergency 
management) 

•  Paul  Liberty  and  Jim 
Greif  (public  relations) 

•  George  Ginkovsky 
(university  police) 


MITRE 

•  Jim  Dear  (Project 
Lead,  NCEL) 

•  Jackson  Ludwig 

•  Jennifer  Mathieu 

•  Alaina  McCormack 

•  Tobin  Bergen-Hill 

•  Karina  Wright 

•  ...  and  many  others 


UNIVERSITY 


and  all  the  SIMEX  participants 
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